Image signal processing

ABSTRACT

An image signal is processed by deriving measurements representing the luminance of a signal; calculating values relating to the local mean, the local standard deviation, the local maximum and/or the local minimum; and computing therefrom local standard coordinates such as z-scores which are independent of brightness and contrast.

The present invention relates to systems and methods for signalprocessing, in particular image signal enhancement and/or signalcompression.

Signal processing systems have the problem that they need to operate inconditions where the signal units are a priori unknown. In terms ofimages, we might observe a boy sitting under a shaded tree such that thedifference in physical units between sunlight and shade is 100000 to 1.Yet, we do not have the capacity to work with such a large signal range.

Embodiments of the present invention are based on the proposal thatstandard coordinates are particularly useful in imaging. Let I(x,y)denote an image brightness at location x and y. The logarithm of theimage response is denoted i(x,y). The brightness and contrast of I canbe changed by a linear transform if i: i(x,y)−mi(x,y)+b where m and bare scalings. The brightness term b scales the brightnesses in an image(b is a multiplicand in non-log space) and accounts for changes in theoverall brightness of a scene. In terms of a trichromatic camera wherethe signal is composed of red-, green- and blue-records we have 3images; R(x,y), G(x,y) and B(x,y) and there might be 3 individualbrightness factors. For example the move to a bluish light from ayellowish one might be modelled by large blue- and small red-brightnessfactors. The contrast term m accounts for the number of log-unitsavailable. Consider taking a picture outside where the signal range is100000 to 1. In contrast a photographic reproduction might have a signalrange of 100 to 1, a shift in log-units of 5 to 2. Coding an image interms of standard coordinates calculated for i(x,y) e.g.$\frac{{i\left( {x,y} \right)} - {\mu\left( {i\left( {x,y} \right)} \right)}}{\sigma\left( {i\left( {x,y} \right)} \right)}$(μ and σ denote mean and standard deviation)has two main advantages. First, it is invariant to μ and σ (so thez-scores for the photographic image and the outdoor signal are thesame). This invariance might be useful for recognising signal content.Second, an image is recast in a way which makes sense in terms of ourown visual perception. We as people see scenes with high dynamic ranges(100000 to 1) yet such a high dynamic range is not used in the cortex.Rather, areas in an image with widely different signal ranges arerecoded into the same units. For example, the physical units in shadowsand highlight regions are small and large respectively. Yet, if theimage is coded in (local) standard coordinates the shadow units willbecome relatively bigger and the highlight coordinates relativelysmaller. The import of this is that we can see into shadows and intohighlights. This tallies with our experience as human observers.Recoding images in terms of standard coordinates provides an elegantsolution to a signal-processing problem with which our own visual systemmust contend.

Aspects of the present invention seek to provide improved methods ofhandling signals. In particular, aspects of the present invention seekto provide a method for signal enhancement by standard coordinates.

According to a first aspect of the present invention there is provided amethod of processing an image signal comprising deriving measurements ofan input image signal including at least a measurement representing theluminance (1) of the signal, characterised in that the method comprisesthe further steps of calculating two of the following: the local mean,the local standard deviation, the local maximum and the local minimum ofsaid measurements, computing therefrom local standard coordinates whichare independent of brightness and contrast, and forming an output imagesignal from the standard coordinates.

In a preferred method the standard coordinates are the local z-scoreswhich have been found to give the best results.

Saturation may be preserved or not-preserved.

The output images may be colour or grey-scale.

According to a second aspect of the present invention, there is provideda device for processing an image signal comprising means for derivingmeasurements of an image signal including at least a measurementrepresenting the luminance of the signal; means for calculating two ofthe following: the local mean, the local standard deviation, the localmaximum and the local minimum; and means for computing therefromstandard coordinates which are independent of brightness and contrast.

Preferred embodiments of the present invention will now be described, byway of example only, with reference to the accompanying drawings, ofwhich:

FIG. 1 is a temperature distribution histogram useful in explaining thepresent invention;

FIG. 2 is a z-score histogram also useful in explaining the presentinvention;

FIGS. 3 to 5 show the steps of a method in accordance with a firstpreferred embodiment of the present invention;

FIG. 6 shows some of the steps of a method in accordance with a secondembodiment of the present invention; and

FIG. 7 shows the corresponding steps of a method in accordance with athird embodmnet of the present invention.

First, let us imagine the following experiment. Three scientists measurethe temperature of a substance as various forces and stresses areapplied to it. Data is compiled which simplistically measurestemperature against amount of stress applied (we might imagine thathigher stresses would lead to higher temperature) where the same stressis applied many times. The scientists now wish to compare their results.But, there is a problem: scientist-1 has measurements in Kelvin,Scientist-2 in Celsius and Scientist-3 in Fahrenheit. How can thescientists compare their data? Of course, the answer is simple: we lookup a reference book and find the formulae that map temperatures acrossscales. For example, Fahrenheit is mapped to Celsius according to:C=0.56F-17.78 and Kelvin to Celsius: C=K-273.2. Note that the correctionin each case is linear. Let us now consider linear corrections in termsof images (but the reasoning extends to general digital signals). Let(R,G,B) denote the red, green and blue value recorded in a digitalimage. These values typically drive a reproduction process such amonitor or a digital printer. It is well known that, to a goodapproximation, many colours are adequately represented by the amounts ofred, green and blue present (see reference [1]). It is important torealise that the RGBs that drive picture creation are oftenrepresentations of the world (e.g. a digital image). Moreover, inreproduction we often make-adjustments to the image so they ‘look’better or have particular signal properties. These two observations(that the signal is a measurement of the world and we might change thesemeasurements) are important considerations. Suppose for example that wetake a picture of a white surface under a whitish light. Thecorresponding RGB=(1,1,1) (white is equal redness, greenness andblueness). Now we alter the illuminant so it is yellower and now record(2,2,1) which is a yellowish colour. Mathematically we might summarisethis as:1

2*=21

2*=21

1*=1  (1)Remarkably, the multiplicands 2,2 and 1 in (1) will map all imagecolours from whitish to yellowish light (see reference [2]). In generalRGBs can be mapped across illuminants according to:R

ρRG

γGG

βR  (2)

Each scaling factor controls the brightness of image measurements.

Let us now imagine that we take a picture of a scene where there is apronounced shadow. When we view the picture on a monitor we discoverthat we cannot see any detail in the shadow: everything is too dark.Assuming that the signal that drives the monitor is between 0 and 1 wecan stretch the dark values (at the cost of compressing the brightvalues) by applying a power function less than 1 (say α):R

R^(α)G

G^(α)B

B^(α)  (3)

Moreover, each color channel might have a separate power term applied.It useful to think of the power term as changing the contrast in animage (the relative importance of bright and dark values). Combining (2)and (3) we see that:R

ρR^(α)G

γG^(α)B

ρβB^(α)  (4)

To first approximation, two images of the same scene but captured withrespect to different viewing conditions and altered to drive differentreproduction processes or to account for personal colour preference willbe related according to (4). The easiest way to change the look and feelof an image is to change its brightness and contrast. As stated therelations in (4) are non-linear. However, taking logarithms of bothsides reveals a natural linear structure:1n R

1n+ρ+α1n R1n G

1n+γ+α1n G1n B

1n+β+α1n B

In principle, Equation (5) corresponds to the temperature experimentdescribed above

The RGBs on the left hand side of are analogous to measurements made byone scientist and those on the right hand side are measurements made bya second scientist. If we know what we are measuring then it is easy totransform coordinates from one side to the other. Indeed, in the colourworld it is now possible to calibrate (map coordinates to a referencemeasurement space) colour devices such as cameras, scanners, printersand monitors (see reference [3]). One implication of this is that if weknow how to produce a good looking image for a reference set of viewingconditions, it should be possible, in principle, to produce a goodlooking image for all conditions. We simply map our image to thereference conditions. Indeed, in a studio environment we can follow thismodus operandi and take a picture of a known reference chart, such as aMacbeth Colour Checker (see reference [4]). We then map the colour chartRGBs so that they look ‘correct’. Applying the same correction to therest of the picture should deliver a good-looking image. Of course theprocess is in fact more complex than that. For example, it would beincorrect to remove the reddish appearance of colours at sunset (inorder to remove colour bias due to illumination) because people like tosee the sunset colours. Nevertheless, the idea that one can calibrateusing reference information is widely used in color imaging and willprovide satisfactory results much of the time.

The question arises of how to proceed if calibration is not possible.Returning to the scientists and their temperature measurements, supposeit transpires that not only are the units that were measured differentbut the equipment that was used was unreliable. Specifically, each ofthe three scientists used measurement devices that had been lying aroundthe lab for some time and the accuracy of their measurements could notbe guaranteed. To a first approximation the measurements (whether inFahrenheit, Celsius or Kelvin) are a linear combination from where theyought to be, though the linear combinations are not known. For exampleone might write: TrueCelsius=A*MeasuredCelsius+B where A and B are notknown. Of course by measuring the temperature of known physicalprocesses (e.g. the boiling and freezing point of water) we could solvefor A and B. If the scientists carried out their experiments some timeago, there is no guarantee, should they calibrate their instruments now,that their calibration would have any bearing on their previous results.Even with such unfavourable, something can be done assuming that thescientists wish to answer particular kinds of questions. Suppose thescientists are interested in determining whether histograms of recordedtemperatures (in each of the three experiments) have the same shape.FIG. 1 shows the temperature distribution in Celsius for two of theexperiments (suppose one set of measurements were made in Fahrenheit andthen converted to Celsius). Clearly the histograms look similar inshape.

Moreover, the main difference between them would appear to be a smallshift and a small stretching. To see if this is the case, we can map thetemperature measurements to so called z-scores (one example of astandard coordinate). Let us linear transform temperature according to:C=aT+b  (6)Let us choose a and b such that the mean of C i.e. (μ(C)) is equal to 0and the standard deviation of C i.e. (σ(C)) is equal to 1. The mean andstandard deviation of a set of N measurements are equal to:$\begin{matrix}{{{\mu(C)} = \frac{\sum\limits_{i = 1}^{N}C_{i}}{N}},{{\sigma(C)} = \sqrt{\frac{\sum\limits_{i = 1}^{N}\left( {C_{i} - {\mu(C)}} \right)^{2}}{N}}}} & (7)\end{matrix}$where it is understood that standard deviation measures the spread of adistribution from the mean (a large standard deviation implies a largespread of measurements and a small standard deviation a small spread).It is straightforward to show that $\begin{matrix}{{a = {{\frac{1}{\sigma(C)}\quad b} = \frac{\mu(C)}{\sigma(C)}}}{{and}\quad{so}}} & (8) \\{C = \frac{\left( {T - {\mu(C)}} \right)}{\sigma(C)}} & (9)\end{matrix}$

FIG. 2 shows the two distributions shown in FIG. 1 where thetemperatures have been mapped to standard coordinates according to (9)(where each distribution is mapped separately). Thus FIG. 2 indicatesthat the histograms of Z-scores for the two data sets are the same.(This indicates that the underlying distribution of data differs by alinear transform)

Clearly the distributions are the same and so some inferences can bemade e.g. the number of small and large measurements are the same forboth experiments. However, it is important to realise that mapping toz-score standard coordinates is not a panacea since it is not and cannotbe as powerful as carrying out a full calibration. It is easy todemonstrate this. Simply take 10 numbers and calculate the standardcoordinates according to (9). Now add a second 10 numbers to thedistribution and recalculate the standard coordinates. Now, because themean and standard deviation of the data will have changed so too mustthe standard coordinates for the original 10 numbers in the dataset.Returning to the image example we can see that equation (5) states thatthe log RGBs in an image shift linearly according to brightness andcontrast. From the foregoing discussion it should also be clear that thestandard coordinates calculated for image RGBs are independent ofbrightness and contrast. Denoting 1n X as x $\begin{matrix}{\frac{\left( {r - {\mu(r)}} \right)}{\sigma(r)} = \frac{\left( {\rho + {\alpha\quad r} - {\mu\left( {\rho + {\alpha\quad r}} \right)}} \right)}{\sigma\left( {\rho + {\alpha\quad r}} \right)}} & (10)\end{matrix}$

The present invention is based on the realisation that (10) is a usefulrepresentation for image processing. Not only have we discarded unknowns(brightness and contrast) but we have done so in way which is intuitive.Let us consider the challenges we ourselves face when we look out at theworld. When we look at a scene we are confronted by the same imageprocessing challenges that a camera faces. For example, we may see a boysitting under a shaded tree on a sunny day.

The physical signal reaching our eye is on the order to 100000 to 1(bright sunlight to shade). Yet, there is no evidence that such a widedynamic range is encoded in the visual cortex. Rather we need to adoptstrategies for encoding images with a smaller dynamic range. One way todo this, in accordance with embodiments of the present invention, is torecode RGBs in terms of standard units. That is we transform imagemeasurements so they are independent of brightness and contrast, i.e. sothey are normalised. In general the normalisation will be carried outlocally. It can be demonstrated that this makes intuitive sense. Abright pixel seen in the context of other bright pixels may be coded assay 3 standard units above the mean. Similarly, a dim pixel seen in thecontext of other dim pixels may also be 3 standard units above the mean.That is, two very different physical measurements may, in principle, becoded as the same number and appear equally bright after coding. Inpractice, standard deviations are calculated across an image withproximate pixels being more important than those far away. So long asthe weighting functions are chosen sensibly, bright areas of the imageoverall appear brighter than darker ones, though detail is drawn out inthe shadows and the highlights.

In embodiments of the present invention, the signal is interpreted ascomposites of measurements made by different devices (though, only onedevice is actually used). We now consider that the difference betweenmeasurements is that the units used are different and unknown. Byconverting all measurements to standard coordinates we effectivelyenforce all devices to work in the same units. In terms of images thisprocess enables us to account for changes in image brightness, dynamicrange and contrast. The 100000 to 1 signal can be recoded to (say) 100to 1 such that we can clearly see the detail of the boy in the shade andalso objects in direct sunlight. We can do this because locally in theimage we are now using the same units.

We interpret a signal as measurements made according to one or morelogical measurement devices. If there are many devices it is assumedthat each is concerned with a local connected part of the signal. It isassumed that the units each device measures differ across time, and ifthere is more than one device per signal, across the signal. However, itis assumed that all measurement units are related to one another bylinear transforms. For a given set of measurements, standard coordinatesare defined by a linear combination of the measurements. For example,Z-scores (one particular standard coordinate) are calculated bysubtracting the mean from the measurements and dividing by themeasurements' standard deviation. Standard coordinates have the propertythat they are invariant to linear transforms. By calculating standardcoordinates, the signals measured (either across time or location) aretransformed to the same units.

Coding images ready for display is a huge research and applied problem(See for example the proceedings of the colour imaging conference foryears 1993 through 2002). Finding the ρ, γ and β, scalars of Equation(2) is usually called colour constancy and this alone has spawnedhundreds of papers; (see references [5,6]) for a review of the majorapproaches. Determining the contrast is more problematic and their arefew papers which explicitly set out to recover the power term a in (5).Recently, the inventor has considered how power term invariance might beachieved, see reference [7].

The questions of contrast and brightness are often implicit inalgorithms for image enhancement and dynamic range compression. Thesealgorithms like (5) tend to work in log space but to either considerbrightness or contrast but never both. To understand how thesealgorithms work it is useful to define some notation and two simpleimage transformations.

First let r_(k)(x,y) denote the log response at location (x,y) in animage for colour channel k=R,G,B. Second, let us define a coordinatetransform from rgb to Luminance 1(x,y), Red-Greeness rg(x,y) and YellowBlueness yb(x,y):I(x,y)=r _(R)(x,y)+r _(G)(x,y)+r _(B)(x,y)rg(x,y)=r ₁(x,y)−r _(G)(x,y)yb(x,y)=r _(R)(x,y)+r _(G)(x,y)−(2*r _(B)(x,y))  (11)Equations similar to (11) appear throughout the colour literature (seee.g. references [1,8]). They are the so-called ‘opponent’ colourchannels: l is a measure of white-black (loosely brightness), rg is ameasure of red and green (in opposition) and yb is a measure ofyellowness blueness (in opposition). Two technical observations areimportant here. First, in the discussion below the above equations arereferred to as opponent channels. Yet there are many ways of calculatingopponent channels (the precise equations above may not be used in themethods cited). Second, (11) is a set of linear equations and as such itis possible to compute r, g and b from l, rg and yb (the equations arereversible). Equation (11) is useful because it allows the separationbetween the achromatic or Luminance (l) and chromatic signal (rg,yb) ofa colour (if we multiply an RGB by a scalar x only l will change). Somealgorithms for dynamic range compression seek only to change theachromatic aspect (l) keeping the chromatic aspect unchanged.

In accordance with a first prior art method, subtracting the meancancels brightness. From (5) we see that a change in brightness ismodelled by an additive term in log space. Since all pixels have thesame additive shift it follows that the mean colour response is alsoperturbed by this shift. As such, subtracting the mean removesdependence on brightness. Denoting a( ) as a function which calculatesthe local mean (or average) it follows thatr _(k)(x,y)−α(r _(k)(x,y))  (12)is independent of brightness.

In accordance with a second prior art method, subtracting the maximumcancels brightness. From (5) we see that a change in brightness ismodelled by an additive term in log space. Since all pixels have thesame additive shift it follows that the maximum colour response is alsoperturbed by this shift. As such subtracting the maximum removesdependence on brightness. Denoting M( ) as a function which calculatesthe local maxima it follows that the expressionr _(k)(x,y)−M(r _(k)(x,y))  (13)is independent of brightness.

Some previous algorithms for dynamic range compression/image enhancementwill now be discussed.

1) NASAs Multiscale Retinex (MSR)

This algorithm (see reference [9]) discounts local brightness in the R,G and B colour channels. Expression (12) above is used with the localaveraging operator a( ) defined as the sum of three Gaussians (givingquite a smooth averaging operator). Contrast is not explicitlyconsidered, though certain global adjustments are made.

It may be noted that, because the local average is subtracted it iscommon for images to look desaturated (or greyish). This problemresulted in an ad hoc (and unsatisfactory) resaturation phase. Examplesof MSR performance can be seen at www.truview.com.

2) Luminance Modified MSR

This algorithm discounts local brightness in the l achromatic channelwith the chromatic channels rg and yb undergoing a global adjustment(see reference [10]). Expression (12) above is used with the localaveraging operator a( ) defined as the sum of three Gaussians (givingquite a smooth averaging operator). The same filtering was used as forthe MSR case.

It may be noted that there was no real improvement observed over theoriginal MSR method. Indeed, the authors of this algorithm suggestedthat a global adjustment might work equally well.

3) Retinex

Edwin Land and John McCann have developed many variants (e.g seereferences [11, 12, 13]) of this algorithm over many years. Inprinciple, expression (13) describes their approach. However, theirdefinition of local maximum is highly complex (being non-linear anditerative). When Retinex works well it can generate pleasing images.Compared to MSR, Retinex works better as subtracting the local maximatends to keep colours looking as they should look (there is nodesaturation problem as with MSR). Unfortunately, when Retinex fails ittends to do so quite badly.

It may be noted that no real mention of the power function is made,although, there is the need to render the output values by anappropriate look up table.

4) Modified Retinex

Because the local maximum function is complex there are many variantswhich provide variants. Worth mentioning are the works of Brainard andWandell (see reference [14] which presents a framework where expressions(12) and (13) might be used), the algorithmic work of Funt et al (seereference [15] which provided the first ‘accredited’ algorithm) andrecent work by Sobel and McCann (see reference [16] which modifies thedefinition of local maxima to take account of overly large local changesin an image). One of the advantages of Retinex is that it is based onpsychophysical observations made with human observers. More recent workby Spitzer (see reference [17]) has carried on this approach and haspresented an operational model of visual processing that appears tosolve some brightness and contrast problems. However, that approach isbased on what is known about visual processing and not on what theproblem is that needs to be solved. Indeed, that is not discussed atall. Spitzer's discloses a method in which computation is carried out onall three opponent channels. Rather speculatively, Land also suggested aRetinex type computation might be carried out in opponent channels (seereference [18]).

It may be noted that Retinex still has problems. The problem it istrying to solve has never been clearly articulated. Good results areonly possible with user intervention. All retinex algorithm have freeparameters which must be ‘tweaked’ on a per image basis.

5) Non-Linear Masking:

In departure from previous methods Moroney (see reference [19]) focuseson contrast and not brightness. In principle, if one divides a logresponse by its local averagel(x,y)/αa(l(x,y))  (14)then contrast must cancel (if we change the contrast in an image thedenominator and numerator are scaled by the same factor which cancels).The definition of a here is a Gaussian but with a very large standarddeviation (so that local values are only given slightly more weight thanfar away values). It may be noted that calculation is carried out in aLuminance channel with the chromatic signal left unchanged. Moreover, asa technical note the Local average is calculated in a slightly differentway than presented above (though for our purposes the detail is notimportant)

It may also be noted that non-linear masking works surprisingly well. Inpart it does so because it is the least aggressive (choosing a Gaussianwith a large standard deviation as an averaging filter means thedefinition of contrast is more global than local). However, it saysnothing about brightness and so cannot account for simple artifacts suchas colour.

In systems and methods according to embodiments of the presentinvention, an image is interpreted as measurements made by different butunknown measurement devices (different coherent parts of the image aremeasured by different devices). This is a logical assumption only. Animage is frequently captured by a single device. But, because we maywish to change the image units in different parts of the image (forexample to alter the dynamic range) it is useful to think of the imageas being a composite of measurements from a set of devices. It isassumed that the units each device measures differ from one another by alinear transform. Local linear corrections are made to the imageeffectively calculating standard coordinates across the image. Localstandard coordinates are independent of brightness and contrast and soare resilient to these changes in an image. Brightness and contrastchanges effectively account for most of the differences between imagesof the same scene. Further, coding images in standard coordinatespresents a plausible solution to an imaging problem that our own visualsystem must face.

Standard Coordinates

Let measurements be denoted x_(i)(i=1,2, . . . ,n) X={x₁, x₂, . . .,x_(n)}. These coordinates are transformed linearly: y_(i)=αx_(i)+b,y={y₁, y₂, . . . y_(n)}. Standard coordinates defined on X and Y are thesame. Numerous standard coordinates (a standard coordinate is a lineartransform which renders the data independent of brightness and contrast)might be calculated:Z-Scores: $\begin{matrix}{S_{i} = \frac{x_{i} - {\mu(X)}}{\sigma(X)}} & (15)\end{matrix}$Max-Min Score: $\begin{matrix}{S_{i} = \frac{x_{i} - {\min(X)}}{{\max(X)} - {\min(X)}}} & (16)\end{matrix}$Min-Mean Score: $\begin{matrix}{S_{i} = \frac{x_{i} - {\min(X)}}{\mu(X)}} & (17)\end{matrix}$

To see that (15) works, let X=10, 20, 30, 40 and Y=10, 30, 50, 30(y_(i)=2x_(i)−10). Calculating (15) for X or Y leads to the set: −1.162,−0.3873, 0.3873, 1.16. Calculating (16) for X or Y leads to the set: 0,0.333, 0.667, 1. Calculating (17) for X or Y leads to the set:0,0.5,1,1.5. Of course there are many other standard coordinate systems.The advantage of scores (15) to (17) is that they are based on simplewell understood statistical operators: mean, max and min. However, thereare many equations of the form 15 to 17 which will remove brightness andcontrast; these are just meant to be examples of “standard co-ordinates”not an exhaustive list.

Calculating Local Standard Coordinates

We are assuming that an image can be thought of composed of a set ofdifferent regions and that each region has measurements in differentunits. It follows if we can standardise coordinates locally.

Calculating the Local Mean

This is easy to do and the literature is replete with possiblealgorithms. The most common method is to convolve an image with anaveraging filter such as a Gaussian. A Gaussian has large values nearthe origin and these decrease monotonically as a function of distancefrom the mean. To illustrate how the convolution works let[1,2,2,2,5,5,5] be a 1-d signal and [0.5, 0.5] an averaging filter. Wesimply place the averaging filter on top of the signal (at a givenlocation), multiply the filter and the signal and sum up. This isrepeated for each position along the signal. Operating in this way wesee that the output of the convolution is [1.5, 2, 2, 3.5, 5, 5]. Weneed to make a couple of comments. First, notice that the edges (1 to 2and 2 to 5) have been diminished in scale. This is as we would expect ifwe locally average the image. Second, we began with 7 inputs but haveonly 6 outputs. The reason for this is that we have not defined what wedo at the boundary of the signal. In terms of 2-dimensional signals, wemight assume that outside of the image everything is zero (these arecalled Dirichlet boundary conditions (see reference [20]). Or that thesignal is not is constant at the boundary (Neumann conditions (seereference [20]). What needs to be done relative to each assumption iswell understood (and is a detail). We implicitly assume this issue hasbeen dealt with in the discussion below. If s(x,y) is a 2-dimensionalsignal (such as an image) and α(x,y) an averaging filter, thenconvolution is defined mathematically as: $\begin{matrix}{{{\mu\left( {x,y} \right)} = {\int_{u}{\int_{v}{{a\left( {u,v} \right)}{s\left( {{x - u},{y - v}} \right)}{\mathbb{d}v}{\mathbb{d}u}}}}}{{{where}\quad{it}\quad{is}\quad{assumed}\quad{\int_{u}{\int_{v}{{a\left( {u,v} \right)}{\mathbb{d}v}{\mathbb{d}u}}}}} = 1}} & (18)\end{matrix}$

If s(x,y)=k (the same value at all locations) then the average of thissignal should be 1.

Calculating the Local Max

A possible way of determining a local maximum is that we could compute alocal maximum say amongst all pixels within 5 pixels from location (xy).We could denote this M₅(x,y). A maximum within a distance 10 could alsobe calculated: M₁₀(xy). In general we might define a local maximum as:$\begin{matrix}{{M\left( {x,y} \right)} = \frac{\sum\limits_{i = 1}^{N}{w_{i}{M_{i}\left( {x,y} \right)}}}{N}} & (19)\end{matrix}$where w_(i) are weights that decrease as i increases and the sum of theweights is 1.

Note, it is important to realise that (19) is just one of many ways inwhich a local maximum might be defined.

Calculating the Local Min

As for the maximum case, we could define a local minima functionm_(d)(x,y) which returns at (x,y) the minima of all pixels within dpixels from (xy). The local minima for an image would be defined as:$\begin{matrix}{{m\left( {x,y} \right)} = \frac{\sum\limits_{i = 1}^{N}{w_{i}{m_{i}\left( {x,y} \right)}}}{N}} & (20)\end{matrix}$where again the sum of the weights is 1.Calculating Local Standard Deviation

Standard deviation is described in Equation (7). This is the square rootof the average of the squared deviation of the mean. Clearly, we mightcalculate this locally according to: $\begin{matrix}{{\sigma\left( {x,y} \right)} = \sqrt{\int_{u}^{\quad}{\int_{v}^{\quad}{{a\left( {u,v} \right)}\left( {{s\left( {{x - u},{y - v}} \right)} - {\mu\left( {{x - u},{y - v}} \right)}} \right)^{2}\quad{\mathbb{d}v}\quad{\mathbb{d}u}}}}} & (21)\end{matrix}$where the averaging filter α( ) may be the same or different than thatused for computing the mean.Calculating Local Standard Coordinates

In general we just substitute (18) to (21) in scores (15) to (17). Forexample the local definition of z-score is given below: $\begin{matrix}{{z\left( {x,y} \right)} = \frac{{{\mathbb{i}}\left( {x,y} \right)} - {\mu\left( {x,y} \right)}}{\sigma\left( {x,y} \right)}} & (22)\end{matrix}$

Local min-max scores or min-mean scores are similarly simply defined.

EXAMPLES

FIGS. 3, 4 and 5 outline a schema for processing an image. In summary wetake an image, convert to log-opponent coordinates then replace theLuminance signal by z-scores and then form an output image. The resultsare images where the details in bright and dark regions are betterbalanced. Moreover, the images generally look more pleasing. In detailthe steps of this particular embodiment are:

-   1) I(x,y) is an input image item-   2) composed of R(x,y), G(x,y) and B(x,y) (RGB color channels)-   3) taking logarithms gives: r(x,y), g(x,y) and b(x,y)-   4) we compute opponent responses according to Equation (11)-   5) we compute local estimates of the mean and standard deviation of    the Luminance channel, (18) and (21); these steps relate to the    weighting functions mentioned previously.-   6) this is used to compute a z-score (22). Of course z-scores will    have some negative and some positive values. To regard z-scores as    luminance signals in the log domain we have to make z-scores all    negative. This is easily achieved in that we simply calculate the    largest z-scores over the whole image and then subtract this from    each individual z-score (at each pixel). After this operation the    z-scores will all be negative with a maximum of 0. If standard    co-ordinates are calculated according to ˜(16) or ˜(17) then they    will be all positive. Again subtracting by the global maxima will    result in all negative values that can be regarded as log brightness    values.    Notice the red-green and blue-yellow channels have their contrast    but not brightness adjusted (by the sigma for the luminance    channel). Empirically it is found that to get good looking images    requires similar contrast in each of the three channels.-   7) We invert (11) and solve for log R, G and B-   8) We invert the logarithm and the images are ready for display.

While the other method steps given above will generate images wheredetail is drawn out in the images, it is the recoding of the data asz-scores in the present example (and standard coordinates in general) insteps 5 and 6 which has significant advantages. The other method stepsare however important especially if the aim is to produce pleasingimages for display. However, the number of ways we might calculateopponent channels, standard coordinates etc are too many to enumerate. Auser will select the other method steps, i.e. 1 to 4, 7 and 8, asappropriate. The common enabling step, however, is the standardcoordinate calculation. When processing input images such as stillphotographs according to the schema described, grey-scale output imagesare obtained which enable detail to be pulled out (e.g. out of shadowregions). If saturation is defined as the angle between RGB and thevector for white, this angle changes pre and post z score calculation.

FIG. 6 shows the corresponding steps 3 to 8 of a second embodiment whichcomputes the min-max score S as given by equation 16.

FIG. 7 shows the corresponding steps of a third embodiment whichcomputes the min-mean score S as given by equation 17.

We emphasise again that the key step is calculating the standardco-ordinates for the luminance image and then reintegrating thisluminance information with the original image to get output RGBs. Ratherthan explicitly calculating the rg and yb opponent channels we mightinstead proceed as follows:

-   1. Let us denote log luminance (as before) 1(x,y)-   2. Let the new log luminance (calculated in step 6 above) be denoted    o(x,y)-   3. Exponentatiating these values gives non-log luminance    L(x,y)=exp(1(x,y)) and O(x,y)=exp(o(x,y))-   4. We then change the RGB for the original image according to the    following 3 equations:    Rnew(x,y)=R(x,y)*O(x,y)/L(x,y)    Gnew(x,y)=G(x,y)*O(x,y)/L(x,y)    Bnew(x,y)=B(x,y)*O(x,y)/L(x,y)

The steps 1 to 4 above are also discussed in reference (10) and resultin colour images where the details are better balanced throughout theimage.

We also note that while we find pleasing images result in computingluminance standard co-ordinates, it is also possible to calculatestandard co-ordinates separately for r(x,y), g(x,y) and b(x,y).

Empirically it is found that the precise definition of Luminance has animpact on the appearance of the image. An alternative inversion (in step7) where (11) is inverted with the constraint that saturation ispreserved (saturation is calculated in RGB and the input saturationequals the output saturation) leads to improved grey-scale images. Ifthe angle defined in the previous paragraph is held fixed, saturationmay be preserved. The computation may be carried out in Luminance only,and the new Luminance integrated to form a colour image.

REFERENCES

-   [1] R. W. G. Hunt. The Reproduction of Color. Fountain Press, 5th    edition, 1995.-   [2] G. D. Finlayson, M. S. Drew, and B. V. Funt. Spectral    sharpening: sensor transformations for improved color constancy. J.    Opt. Soc. Am. A, 11(5):1553-1563, May 1994.-   [3] H.R. Kang. Color Technology for electronic imaging devices.    SPIE, 1997.-   [4] C. S. McCamy, H. Marcus, and J. G. Davidson. A color-rendition    chart. J. App. Photog. Eng., pages 95-99, 1976.-   [5] G. D. Finlayson. Coefficient color constancy, 1995. MSc thesis,    Simon Fraser University, School of Computing Science.-   [6] G. D. Finlayson, S. D. Hordley, and P. M. Hubel. Color by    correlation: A simple, unifying framework for color constancy. IEEE    Transactions on pattern analysis and machine intelligence, 23(11):    1209-1221, Nov. 2001.-   [7] G. D. Finlayson and R. Xu. Log gamma normalization. In IS&T 10th    color imaging conference. November 2002. to appear.-   [8] B. A. Wandell. Foundations of Vision. Sinauer Associates, 1st    edition, 1995.-   [9] Zia ur Rahman, Daniel Jobson, and Glenn Woodell. Method of    improving a digital image. U.S. Pat. No. 5,991,456 (23 Nov. 1999),    1999.-   [10] K. Barnard and B Funt. Investigations into multi-scale retinex    (msr). In Colour Imaging Vision and Technology, ed. L. W. MacDonald    and M. R. Luo, pages 17-36. 1999.-   [11] E. H. Land. The retinex theory of color vision. Scientific    American, pages 108-129, 1977.-   [12] E. H. Land and J. J. McCann. Lightness and retinex theory. J.    Opt. Soc. Amer., 61:1-11, 1971.-   [13] J. J. McCann. Lessons learned from mondrians applied to real    images and color gamuts. In IS&T and SD's 7th Color Imaging    Conference. 1999.-   [14] D. A. Brainard and B. A. Wandell. Analysis of the reinex theory    of color vision. J. Opt. Soc. Am., A, 36:1651-1661, 1986.-   [15] B. V. Funt, F. Cuirea, and J. McCann. Retinex in matlab. In    IS&T and SID's 8th Color Imaging Conference, pages 112-121. 2000.-   [16] R. Sobel. Improving retinex algorithm for rendering wide    dynamic range photographs. In Human Vision and Electronic Imaging    VII, volume 4662, pages 341-348. 2002.-   [17] Hedva Spitzer. Method for automatic partial white balance    correction U.S. Pat. No. 5,771,312 (23rd Jun. 1998), 1998.-   [18] E. H. Land. Recent advances in retinex theory and some    implications for cortical computations: Color vision and the natural    image. Proc. Natl. Acad. Sci, 80:5163-5169, 1983.-   [19] N. Moroney. Local color correction using non-linear masking. In    IS&T and SID's 8th Color Imaging Conference, pages 108-111. 2000.-   [20] D. L. Kreider, R. G. Kuller, D. R. Ostberg, and F. W. Perkins.    An introduction to linear analysis. Addison Wesley, 1966.

1. A method of processing an image signal comprising derivingmeasurements of an input image signal including at least a measurementrepresenting the luminance (1) of the signal, characterised in that themethod comprises the further steps of calculating two of the following:the local mean, the local standard deviation, the local maximum and thelocal minimum of said measurements, computing therefrom local standardcoordinates which are independent of brightness and contrast, andforming an output image signal from the standard coordinates.
 2. Amethod according to claim 1, wherein the local mean and the localstandard deviation are calculated and said standard co-ordinates are thelocal z-scores.
 3. A method according to claim 2, wherein, in parallelwith the computation of the z-scores, colour channel signals areobtained by dividing by said standard deviation.
 4. A method accordingto claim 1, wherein the local maximum and the local minimum arecalculated and said standard co-ordinates are the local max-min scores.5. A method according to claim 1, wherein the local minimum and thelocal mean are calculated and said standard co-ordinates are the localmin-mean scores.
 6. A method according to claim 1 wherein, before thecalculating step, logarithms are taken of the R, G and B colour channelvalues and opponent responses are computed.
 7. A method according toclaim 6 wherein, besides luminance, said opponent responses includered-greenness and yellow-blueness.
 8. A method according to claim 1wherein, after computing the local standard coordinates, the logarithmsof the R, G and B values are determined and the determined logarithmsare inverted.
 9. A method according to claim 1 wherein a grey scaleoutput image is obtained and saturation is not preserved.
 10. A methodaccording to claim 1 wherein a colour output image is obtained andsaturation is preserved.
 11. A computer programmed to perform a methodaccording to claim
 1. 12. A device for processing an image signalcomprising means for deriving measurements of an image signal includingat least a measurement representing the luminance of the signal; meansfor calculating two of the following: the local mean, the local standarddeviation, the local maximum and the local minimum; and means forcomputing therefrom standard coordinates which are independent ofbrightness and contrast.
 13. A device according to claim 12 wherein themeasurement deriving means further derives measurements representingred-greenness and yellow-blueness.